General-Reasoner: Advancing LLM Reasoning Across All Domains

We sincerely thank the reviewer for highlighting the contribution of our datasets and the effectiveness of the model-based verifier in RLVR training. We also appreciate the insightful comments and valuable feedback. Below is our point-by-point response addressing each comment raised:

1. The use of a model-based verifier (or reward model in RL) to replace rule-based verification is not novel, as evidenced by prior works like [1][2][3], and has been applied even in math-focused tasks.

   The works mentioned, such as VerifyBench [1] and xVerify [2], are very close concurrent works to ours, published within 3 months of the submission deadline. Their preprints were actually released after the first version of our preprint.

   Beyond timing considerations, our work's main contribution is scaling RLVR data across diverse domains, with the generative model-based verifier serving as a solution that effectively supports this data scaling. This differentiates our approach from previous works (e.g., [3]) that only focus on improving answer verification in the math domain. Our work provides a scalable framework that advancing the field beyond narrow, domain-specific reasoning enhancements.

2. GRPO already employs model-based reward systems

   While model-based reward systems aren't entirely novel, our work specifically focuses on developing a lightweight, generative model-based verifier that enables scaling across diverse domains.

3. The sharp performance rise at step 120 may be coincidental.

   We thank the reviewer for raising this question. To further investigate, we retrained the model-verifier-based General-Reasoner-Qwen3-4B corresponding to Figure 4 for 500 steps, with the results shown below:

   Steps	MMLU-PRO
   40	0.5767
   80	0.5865
   120	0.5958
   160	0.6061
   200	0.6140
   240	0.6082
   280	0.6173
   320	0.6142
   360	0.6239
   400	0.6234
   440	0.6319
   480	0.6312

   This demonstrates that the performance improvement with the model-based verifier is consistent and not coincidental.

4. Expand baseline comparisons: Include more results for Qwen3-14B baselines

   Our main baselines in this work are the ZeroRL approaches such as SimpleRL. However, since Qwen3-14B is a newly released model, no baselines from previous works are available for it. The comparisons with ZeroRL baselines focus on the Qwen2.5 models, while our Qwen3 experiments demonstrate that our data and training framework also bring improvements to these recent stronger models.

   To further enrich the baseline comparison, we trained two SFT baselines: 1) we directly used WebInstruct-Verified data to conduct SFT with questions and extracted short answers, and 2) we used the original WebInstruct (without processing it as verifiable answer data with open answer form) to conduct SFT. The results are shown below:

   Model	Post Training	MMLU-PRO	SuperGPQA
   Qwen3-4B-Base	-	0.516	0.254
   Qwen3-4b-webinstruct-verified-sft	SFT	0.4397	0.281
   Qwen3-4b-webinstruct-sft	SFT	0.4386	0.227
   General-Reasoner-Qw3-4B	RL	0.628	0.325
   Qwen3-14B-Base	-	0.642	0.365
   Qwen3-14b-webinstruct-verified-sft	SFT	0.5332	0.323
   Qwen3-14b-webinstruct-sft	SFT	0.4465	0.2542
   General-Reasoner-Qw3-14B	RL	0.703	0.399

   The results demonstrate that SFT training on strong base models actually degrades performance. In contrast, our RL-based training framework consistently delivers significant improvements over the base model.

5. Separate Qwen3-14B and Qwen2.5 results

   We appreciate your suggestion and will separate the Qwen3 and Qwen2.5 results for the 14B models into different groups to enhance clarity in the presentation.

6. The answer-pair verification approach is limited to QA-style tasks with short answers, failing to generalize to broader tasks like summarization, writing, or translation.

   This work focuses on improving LLM reasoning capabilities specifically within the scope of reinforcement learning with verifiable reward (RLVR) for reasoning-intensive tasks. Tasks like summarization, writing, and translation are not reasoning-intensive and fall outside the scope of our research. We will expand this discussion in the limitations section.

7. typo in OpenReview abstract

   We thank the reviewer for their careful reading. We will fix the typo.

We hope our response has addressed your main questions and concerns. We welcome further discussion during the reviewer-author discussion period.

We sincerely thank the reviewer for highlighting our empirical results and our contribution in scaling data for RLVR training across diverse domains. We also appreciate the valuable feedback and comments. Below is our point-by-point response addressing each comment raised:

1. “What's the SFT data for "Qwen3-4B-Instruct" in the second block of Table 2? Is that just one of the official relasedcheckpoints from Qwen?”

   Yes, the Qwen3-4B-Instruct baseline is one of the official released checkpoints from Qwen. We will add further clarification in the experiment setup.

2. “How exactly was [kept examples where you can find human-written answers] implemented?”

   Thanks for raising this question, here what we trying to say is not guarantee “human” answer in contrast to “synthetic” answers. As the source data is from webcrawling rather than LLM generated, it is naturally “human-written” contents, however, we use Gemini-Pro to ensure that the completeness of human-written answers for the extracted questions. We will update the description to improve the clarity.

3. “a significant portion of the dataset is still math problems (34%). “

   While our dataset encompasses diverse reasoning domains, we maintained a significant portion (34%) focused on math problems. This decision is based on the fact that many STEM-domain tasks depend on mathematical reasoning abilities. Nevertheless, we highlight that 66% of our dataset covers non-mathematical domains, which represents a substantial departure from existing works that focused only on mathematical reasoning.

4. “How do you justify the quality of the final dataset? shouldn't you do some data ablation to justify its importance? “

   In Table 4, we compare performance when using our full training dataset versus a math-only subset while keeping the verifier fixed. The results demonstrate the effectiveness of scaling training data across multiple reasoning domains. Our method also outperforms previous approaches like SimpleRL, which primarily focuses on math-only data, further validating the importance of incorporating diverse reasoning tasks.

5. “insights beyond existing / concurrent works”

   1. Demystifying Long Chain-of-Thought Reasoning in LLMs

      Preprint Date: Feb 5th, 2025 (concurrent work within 3 month)

      We would like to clarify that the WebInstruct dataset used in the referenced work was designed for supervised fine-tuning (SFT). In contrast, our WebInstruct-Verified dataset specifically targets high-quality RLVR training data. To demonstrate the effectiveness of our approach, we conducted additional experiments comparing SFT training with WebInstruct against RL training using WebInstruct-Verified on Qwen3-4B and Qwen3-14B.

      Model	Post Training	MMLU-PRO	SuperGPQA
      Qwen3-4B-Base	-	0.516	0.254
      qwen3-4b-webinstruct-sft	SFT	0.4386	0.227
      General-Reasoner-Qw3-4B	RL	0.628	0.325
      Qwen3-14B-Base	-	0.642	0.365
      Qwen3-14b-webinstruct-sft	SFT	0.4465	0.2542
      General-Reasoner-Qw3-14B	RL	0.703	0.399

      The results demonstrate that using our curated data with the proposed training framework is more effective than applying SFT over WebInstruct. Based on these findings, we conclude the following takeaway:

      Insight: The curation of extract verifiable question-answer pair to create diverse domain RLVR training data is more effective than directly apply on crawled instruction following data such as WebInstruct.

   2. Pitfalls of Rule- and Model-based Verifiers – A Case Study on Mathematical Reasoning

      Preprint Date: May 28th, 2025 (follow up work of ours)

      This referenced work is actually a follow-up study to ours, and notably, our verifier model was extensively employed in their experimental comparisons. Their experiments validate that our high-quality model-based verifier performs well compared to other verifiers. We highly appreciate their additional analyses and insights.

      Our primary contribution is scaling Zero-RL training across general domains and evaluating the broader improvements in reasoning capabilities of LLMs. While an extensive characterization of model-based verifiers is beyond the scope of our current work, especially since their study specifically addresses mathematical reasoning, we aim to clearly deliver this key takeaway:

      Insight: Model-based verifier have important role in help scaling the RLVR training data across diverse domains.

   3. Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning

      Preprint Date: July 1th, 2025 (follow up work of ours)

      Their work includes General-Reasoner in its major evaluation, which demonstrates strong transferability. We find their analyses of transferability particularly impressive.

      Returning to our research, our findings align with evidence that mathematical reasoning enhances general LLM capabilities. However, we demonstrate significant additional benefits from scaling RL training to include diverse reasoning tasks. Comparing SimpleRL-14B with Qwen2.5-14B-Base on MMLU-PRO shows that math-focused RL improves general reasoning performance. Our General-Reasoner-Qwen2.5-14B further surpasses these results, highlighting the substantial gains from broader-domain reasoning training. Additionally, Table 4 demonstrates the improvements achieved by incorporating diverse data compared to using math-only data.

      Insight: While math reasoning enhances general LLM capabilities, scaling training to include diverse reasoning domains significantly increases effectiveness.

   We sincerely thank the reviewer for bringing up these comparisons, which enrich the discussion of our paper. We hope that the above analysis of the suggested comparisons clearly demonstrates the specific contributions and insights our work provides.

We sincerely thank the reviewer for acknowledging our contribution to scaling RLVR across diverse domains. We appreciate the insightful comments and valuable feedback. Below is our point-by-point response addressing each comment raised:

1. “They don’t isolate the RL fine-tuning from the dataset or verifier.”

   In Table 4, we isolate the effect of dataset diversity by fixing the verifier and comparing performance between the full training dataset and a math-only dataset. The results clearly demonstrate that scaling training data across diverse domains improves general reasoning capabilities. Similarly, in Table 5, we isolate the effect of verifier quality by fixing the dataset and comparing our model-based verifier against a rule-based approach. This comparison highlights the advantages of using a model-based verifier for handling diverse domains. Through these controlled ablation studies, we separately examine the impact of dataset scale and verifier quality on model performance.

2. “There is no pure knowledge-distillation / direct SFT (and other no-RL) baseline to isolate the effect of the RL procedure itself.”

   Our primary goal is to evaluate the proposed Zero-RL framework, where the model learns exclusively from questions and verifiable answers without supervised CoT labels. Therefore, our main baselines include Zero-RL methods, such as SimpleRL.

   We compared our approach against models like Qwen2.5-14B-Instruct, which were trained with supervised fine-tuning (SFT). Our model consistently outperforms these models, highlighting the benefits of our RL-based training. This is particularly significant considering our training data consists of questions with verifiable answers but no explicit CoT supervision.

   To further enrich our analysis, we conducted additional experiments:

   • Training models directly with SFT to generate answers without reasoning steps.
   • Training models with the original WebInstruct dataset (without verifiable answer extracted), which provides a relatively direct comparison between CoT-SFT and Zero-RL approaches.

   The following table presents additional experimental results for Qwen3-4B-Base and Qwen3-14B-Base models:

   Model	Post Training	MMLU-PRO	SuperGPQA
   Qwen3-4B-Base	-	0.516	0.254
   qwen3-4b-webinstruct-verified-sft	SFT	0.4397	0.281
   qwen3-4b-webinstruct-sft	SFT	0.4386	0.227
   General-Reasoner-Qw3-4B	RL	0.628	0.325
   Qw3-14B-Base	-	0.642	0.365
   qwen3-14b-webinstruct-verified-sft	SFT	0.5332	0.323
   qwen3-14b-webinstruct-sft	SFT	0.4465	0.2542
   General-Reasoner-Qw3-14B	RL	0.703	0.399

   These results show that SFT on strong base models like Qwen3-4B-Base and Qwen3-14B-Base may even have a negative impact, while RL over our curated dataset consistently improves performance compared to the base model.

3. “Did the authors compare the verifier against Gemini generated answers?”

   Yes, our verifiers were evaluated against Gemini-generated answers. We will add more detailed information about this comparison in Section 5.3 of the manuscript.

4. “How is generative verifier v.s. scalar RLVR.”

   To clarify the effectiveness of our generative verifier compared to scalar verifier, we trained a scalar verifier with the same backbone and training data and evaluated its agreement with the Gemini-based verifier (matching Figure 5). The scalar verifier judges answers based on the probability of Pr(True).

   Answer Type	rule_verifier	scalar_verifier	model_verifier
   Boolean	0.233	0.851	0.882
   Expression	0.192	0.542	0.743
   Float	0.105	0.477	0.755
   Fraction	0.699	0.839	0.924
   Integer	0.460	0.680	0.829
   List	0.065	0.472	0.736
   Matrix	0.515	0.669	0.840
   Multiple Choice	0.336	0.761	0.869
   Other	0.106	0.365	0.681
   Percentage	0.277	0.502	0.683
   String	0.187	0.668	0.826

   These results demonstrate that our generative verifier consistently outperforms the scalar verifier across all answer types. This performance advantage can come from the generative verifier's ability to leverage chain-of-thought reasoning during the verification process.

   In addition to automatic evaluation comparing against the Gemini verifier, we also conducted human evaluation on 200 randomly sampled question-answer pairs to assess the agreement with human judgment as presented below:

   Answer Type	rule_verifier	gemini_verifier	scalar_verifier	general-verifier (ours)
   Boolean	0.750	1.000	1.000	1.000
   Expression	0.516	0.935	0.806	0.806
   Float	0.585	0.892	0.754	0.877
   Fraction	0.750	1.000	1.000	1.000
   Integer	0.852	0.963	0.926	0.963
   List	0.556	0.778	0.778	0.889
   Matrix	0.667	1.000	0.833	0.833
   Multiple Choice	0.773	0.864	0.864	0.909
   Other	0.000	1.000	0.500	1.000
   Percentage	0.600	1.000	0.600	0.700
   String	0.312	0.938	0.812	0.812

   These results further confirm that our general verifier is more effective than the scalar verifier.

5. “Alternative verifiers? What if the verifier is trained only with open LLMs? “

   Our main contribution is scaling Zero-RL training across general domains, with the verifier being a secondary contribution. Future work could explore alternative training strategies or using open-source LLMs for verification, as well as testing various combinations of verifier models. We will acknowledge this limitation and outline these future directions in our manuscript.

We hope our responses have addressed the reviewer's main questions and concerns. We welcome further discussion during the reviewer-author discussion period.

We sincerely thank the reviewer for acknowledging our strengths in innovative dataset creation and reward design, as well as our empirical performance. We also appreciate the insightful comments and valuable feedback. Below is our point-by-point response addressing each comment raised:

1. “Related work on Zero-RL is not acknowledged” (W1, Q4)

   Our work builds upon recent Zero-RL methods, including DeepSeek Zero and SimpleRL, as acknowledged in the Introduction and Related Work (e.g., line 26, line 102). However, our primary contribution differs in both motivation and execution. While existing Zero-RL approaches primarily focus on mathematical reasoning, we explicitly aim to enhance general reasoning across diverse domains by significantly expanding verifiable training data and evaluations. We will further clarify this distinction in the related work section.

2. “Unclear why Zero-RL works at small scale” (W2, Q1)

   Figure 4 presents the training curve for the Qwen3-4B model. The initial rapid gains likely stem from the model aligning its answer formats with expectations, while continued improvements increasingly depend on data quality and verifier effectiveness. We will include more detailed discussions results section.

3. “Potential risk of reward hacking is not addressed” (W3, Q2)

   In the early stages of our experiments, we did observed risks of reward hacking. The policy model tended to inject long content into the latex box for short answers. This resulted in increasing training rewards but poor test performance.

   As mentioned in line 200, our reward design thus includes penalties for answers that significantly deviate in length from the ground truth short answers. This constrains the model's ability to inject hacking content intended to mislead the verifier.

   Our comprehensive evaluation across various downstream reasoning tasks demonstrates that our verifier-guided training produces robust performance improvements, suggesting that the resulting model is not affected by the potential risk.

4. “How robust is the generative verifier to answer formatting” (W4, Q3)

   In Figure 5, we evaluate the agreement between our smaller generative verifier and the Gemini-based verifier, demonstrating high alignment between the two.

   However, we acknowledge the reviewer's concern regarding the assumed correctness of the verifier.

   Regarding the reviewer's example of a chemistry case where "26.32" is marked correct instead of "26.3," this illustrates precisely why we need a model-based verifier that makes judgments conditioned on question context.

   To assess verifier quality, we manually evaluated 200 randomly sampled question-answer pairs, comparing the verification agreement of different verifiers with human verification. Below are the results:

   Answer Type	rule_verifier	gemini_verifier	general-verifier (ours)
   Boolean	0.750	1.000	1.000
   Expression	0.516	0.935	0.806
   Float	0.585	0.892	0.877
   Fraction	0.750	1.000	1.000
   Integer	0.852	0.963	0.963
   List	0.556	0.778	0.889
   Matrix	0.667	1.000	0.833
   Multiple Choice	0.773	0.864	0.909
   Other	0.000	1.000	1.000
   Percentage	0.600	1.000	0.700
   String	0.312	0.938	0.812

   These results demonstrate the high accuracy and robustness of our generative verifier across diverse answer formats. This human evaluation further confirms the significant advantage of our generative verifier compared to traditional rule-based methods. We will incorporate this detailed analysis based on human evaluation into the revised manuscript.

5. “No statistical significance or variance reporting” (W6)

   Our evaluation protocol adheres to established practices in prior research. For high-variance datasets like AIME24/25, we report performance averaged over 32 runs, consistent with methods such as Simple-RL. For comprehensive benchmarks like MMLU-Pro and SuperGPQA, we report averages across large test sets containing more than 10,000 queries each, following the protocols established by the original datasets.

6. “WebInstruct-Verified dataset lacks transparency” (W5)

   We would like to highlight that the WebInstruct-Verified dataset will be publicly released. To further improve transparency, we will also provide the unfiltered dataset and detailed outputs from the Gemini model used in data curation. This additional information will ensure complete transparency in our dataset creation process.

We hope our response has addressed your main questions and concerns. Your suggestions have helped us improve the paper. We welcome further discussion during the reviewer-author discussion period.

We thank the reviewer for carefully reviewing our manuscript. Given that only a few days remain in the reviewer–author discussion period, we kindly ask if our rebuttal has addressed your initial major concerns. If there are any additional comments regarding our rebuttal. We would be very happy to further address them.

We sincerely thank the reviewer for the valuable and encouraging feedback acknowledging our contribution in tackling an important gap in LLM reasoning. Below is our point-by-point response to the comments:

1. “The technical novelty is relative limited, but the main contributions lie in the curated dataset, training paradigm and experiments.” (W1)

   We agree with the reviewer that our primary contributions emphasize providing a comprehensive and scalable framework rather than technical novelty. Specifically, we contribute a diverse, high-quality dataset for RLVR training across multiple domains, alongside a cohesive training paradigm that integrates "verifiable data" with a "small model based verifier." This integration enhances scalability across various data sizes and domains while simplifying the overall approach.

2. “There is a lack of quality control in the collected data.” (W2)

   We acknowledge that the absence of human-based quality control is a limitation in our dataset curation process. Our approach relies on automatic filtering, including dropping questions that Gemini models consistently fail to solve across multiple (8) attempts. This filtering strategy helps ensure that questions in the final dataset are solvable and free from errors. We will add a discussion about implementing further human quality control in the limitations section.