Reproducibility and Use of Standard Public Benchmarks

We appreciate the importance of using standard public benchmarks and have indeed conducted evaluations on several open benchmarks. Our results show:

• Significant improvement in coding tasks (e.g., LiveCodeBench: 30.3 -> 36.2).
• Slight improvements or comparable performance in general knowledge tests (e.g., MMLU: 87.3 -> 87.3, MMLU pro: 79.6 -> 80.1).
• Slightly lower but comparable performance on simple math tasks (e.g., AMC23: 68.8 -> 68.7, MATH500: 88.2 -> 87.8).
• Comparable performance on high-difficulty math tasks (e.g., AIME: 18.6 -> 17.9, 26.9 -> 27.2).

However, we've identified limitations in many public benchmarks:

• Potential IID issues, not fully reflecting real-world challenges.
• Risk of data leakage due to widespread use.
• Insufficient difficulty to comprehensively assess advanced models.

For instance, our model can score nearly 90 points on some math benchmarks, but only around 30 points on our more challenging internal test sets.

Consequently, we developed an internal evaluation suite with the following advantages:

• Higher difficulty, better differentiating top-tier model capabilities.
• Continuous iteration, preventing model overfitting.
• Black-box testing, ensuring fair and authentic evaluation.

We believe combining public benchmarks with rigorous internal assessments provides a more comprehensive and reliable evaluation of model performance. We will continue to monitor and use public benchmarks while refining our internal evaluation system.

Regarding the use of open-source models and datasets, we acknowledge their value in enhancing reproducibility. We are considering conducting supplementary experiments with these resources and will include them in future work to provide a more comprehensive comparison.

Systematic Exploration of Task Prioritization Design

We appreciate the reviewer's request for per-task learning curves for the Early Training Emphasis strategy. Due to our rebuttal policy, we are unable to include new figures in the PDF. However, readers can refer to Figure 4 in the original paper, specifically the first 1400 steps, to gain insights into the learning progression for math and coding tasks.

In addition to the information provided in the paper, we have conducted supplementary experiments regarding the order of training for code and math tasks. On our 25B model, we tested the following configurations:

1. Training exclusively on coding data first
2. Training exclusively on math data first
3. Our current mixed approach

Results on our internal test set V1 showed:

• Coding first: 38.1
• Math first: 37.8
• Current mixed approach: 38.8

These results demonstrate that our current mixed approach outperforms exclusively prioritizing either coding or math tasks.

We hope this additional information provides further clarity on the effects of task ordering in our training strategy. We appreciate the reviewer's suggestion and will consider generating more detailed per-task learning curves for future iterations of our work, as computational resources allow.

Writing and Presentation

We acknowledge the issues of redundancy and lack of clarity in our writing. We will promptly revise the abstract to eliminate redundancy and clearly convey our contributions. Moreover, we'll review and enhance the clarity and conciseness of other paper parts. Thank you for pointing these out.

We present the revised abstract below, incorporating feedback on clarity and scope:

"Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human preferences and values. While recent research has primarily focused on algorithmic advancements—such as reducing computational overhead or strengthening reward models to mitigate reward hacking—the critical role of prompt-data construction and its scalability has received comparatively less attention. In this paper, we address this gap by systematically exploring data-driven bottlenecks that currently hinder RLHF performance scaling, focusing specifically on the challenges posed by reward hacking and decreasing response diversity. To mitigate reward hacking, we introduce a hybrid reward system combining reasoning task verifiers (RTV) and a generative reward model (GenRM). This approach enables accurate assessment of responses against clearly defined ground-truth solutions. Additionally, in order to ensure response diversity and enhance learning effectiveness, we propose a novel prompt-selection method named Pre-PPO, explicitly identifying training prompts that are inherently challenging and thus less prone to reward hacking. Furthermore, we find that prioritizing mathematical and coding tasks during the early phases of RLHF training significantly boosts performance, given that these tasks naturally encode fine-grained response distinctions and possess clearly defined ground truths. Through experiments conducted on both small and large models, we demonstrate the effectiveness and scalability of our proposed methods. Our approach exhibits robust generalization capabilities, especially on challenging and out-of-distribution tasks, while yielding significant improvements in mathematics-intensive (STEM) and coding domains. Moreover, the proposed strategies enable the model to effectively capture subtle, task-specific distinctions during the RLHF process, substantially enhancing overall model performance. This work emphasizes the critical role of careful data construction and provides practical methodologies for addressing key performance bottlenecks in RLHF."

Broader Impact

Coverage of Tasks Requiring Long-Form Outputs:

While the current study primarily focuses on tasks involving short-form outputs, we have initiated experiments targeting long-form content generation. In our forthcoming paper, we employ the same methodological framework to address long-text generation tasks. Initial findings are promising, indicating that our approach scales well to long-cot setting. We believe that integrating these additional evaluations will provide a more comprehensive understanding of the applicability and robustness of our method in generating extensive textual content.

Additional Remarks:

Our approach shares similarities with concurrent work, specifically Deepseek-R1, in several aspects. Both methodologies involve initial training on verifiable tasks such as code and mathematics, followed by a preference learning phase. However, our research uniquely identifies and demonstrates the efficacy of this approach in the context of short-text generation. This distinction underscores the versatility and potential of our method across different text lengths and complexities.

The Individual or Combined Advantages of Using the BT Reward Model, GenRM, and RTV

We appreciate the reviewer's concern regarding the demonstration of individual and combined advantages of the BT reward model, GenRM, and RTV. We would like to clarify that we have indeed conducted a comprehensive comparative analysis of these components in the Appendix under "Further Experimental Analysis." This analysis provides substantial evidence for the effectiveness and distinctions between these methods.

Specifically, our analysis focuses on two critical aspects:

1. Reward Hacking: We demonstrate that RTV is significantly more robust against reward hacking compared to other methods. Our experiments show a clear hierarchy in performance: RTV > GenRM with ground truth > GenRM without ground truth ≈ BT RM. This hierarchy illustrates the superior ability of RTV to maintain integrity in reward signals, crucial for reliable model training.

2. Fine-grained Difference Detection: Our analysis reveals that RTV excels in identifying subtle differences between responses, outperforming other methods. The performance ranking follows the same pattern: RTV > GenRM with ground truth > GenRM without ground truth ≈ BT RM. This capability is essential for nuanced evaluation and improvement of model outputs.

Test Dataset Details and Evaluation Methodology

We apologize if this crucial information was not sufficiently highlighted in the main text. We will consider emphasizing these findings more prominently in the final version to ensure their significance is not overlooked. We would like to offer additional information about our test datasets:

The test sets are independently created by evaluators and are divided into white-box and black-box test collections. The white-box tests are visible to algorithm developers, while the black-box tests are not. The domains covered in these datasets are already listed in the paper.

Furthermore, these test sets are updated every 1-2 months. During each update, new challenging questions are added, and some simpler questions are replaced to maintain the overall difficulty level. This regular refresh ensures that the evaluation remains relevant and challenging over time.

V1.0 and V2.0 refer to two consecutive versions of these test sets. This versioning reflects our commitment to continuous improvement and adaptation of our evaluation metrics.

In the final version of the paper, we will include a dedicated section that elaborates on these key aspects of our experimental setup, providing readers with a more comprehensive understanding of our evaluation process and the robustness of our results.

Theoretical Justification and Novelty

While there isn't much theoretical analysis, in the "Further Experimental Analysis" appendix, we've made some relevant analyses and interpretations.

1. Analyzing prompt filtering by Pre-PPO strategy We collected five responses per prompt, calculated the maximum edit distance among them, and categorized prompts into bins. Then we computed the average normalized reward model score for each bin. Edit distance can reflect the granularity of response differences (larger for coarser-grained, smaller for finer-grained).
2. Observations from the analysis Task-type differences: For tasks supervised by GenRM with ground truth (e.g., math and logic) and without ground truth (e.g., creative writing and cosplay), the normalized reward model scores show different trends as edit distance changes. The model captures coarse-grained differences easily for tasks without ground truth, and is more sensitive to fine - grained ones for tasks with ground truth. Effect of Pre-PPO exclusion: In the Pre-PPO strategy, we excluded certain prompts. An ablation study found that reintroducing previously excluded math and logic prompts improved overall performance marginally. This implies that learning coarse-grained patterns from creative writing and cosplay tasks can harm the scalability of RLHF data. Hypothesis on early task emphasis: Emphasizing math and coding tasks in early training may help the model capture fine-grained distinctions first, reducing the negative impact of learning coarse-grained patterns too early.
3. Reward model analysis Analyzing reward score differences across prompt bins, we found that GenRM with ground truth and RTV assign larger score differences in bins with smaller edit distances. GenRM without ground truth fails to show meaningful score differences in these bins. RTV shows greater ability to capture subtle response distinctions compared to GenRM with ground truth at low edit distances. Overall, reward models with ground truth or verification feedback are more sensitive to fine-grained response variations.

Experimental Design and Clarity

We appreciate the reviewer's request for per-task learning curves for the Early Training Emphasis strategy. Due to our rebuttal policy, we are unable to include new figures in the PDF. However, readers can refer to Figure 4 in the original paper, specifically the first 1400 steps, to gain insights into the learning progression for math and coding tasks.

In addition to the information provided in the paper, we have conducted supplementary experiments regarding the order of training for code and math tasks. On our 25B model, we tested the following configurations:

1. Training exclusively on coding data first
2. Training exclusively on math data first
3. Our current mixed approach

Results on our internal test set V1 showed:

• Coding first: 38.1
• Math first: 37.8
• Current mixed approach: 38.8

These results demonstrate that our current mixed approach outperforms exclusively prioritizing either coding or math tasks.

We hope this additional information provides further clarity on the effects of task ordering in our training strategy. We appreciate the reviewer's suggestion and will consider generating more detailed per-task learning curves for future iterations of our work, as computational resources allow.

Writing and Presentation

We acknowledge the issues of redundancy and lack of clarity in our writing. We will promptly revise the abstract to eliminate redundancy and clearly convey our contributions. Moreover, we'll review and enhance the clarity and conciseness of other paper parts. Thank you for pointing these out.

We present the revised abstract below, incorporating feedback on clarity and scope:

"Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human preferences and values. While recent research has primarily focused on algorithmic advancements—such as reducing computational overhead or strengthening reward models to mitigate reward hacking—the critical role of prompt-data construction and its scalability has received comparatively less attention. In this paper, we address this gap by systematically exploring data-driven bottlenecks that currently hinder RLHF performance scaling, focusing specifically on the challenges posed by reward hacking and decreasing response diversity. To mitigate reward hacking, we introduce a hybrid reward system combining reasoning task verifiers (RTV) and a generative reward model (GenRM). This approach enables accurate assessment of responses against clearly defined ground-truth solutions. Additionally, in order to ensure response diversity and enhance learning effectiveness, we propose a novel prompt-selection method named Pre-PPO, explicitly identifying training prompts that are inherently challenging and thus less prone to reward hacking. Furthermore, we find that prioritizing mathematical and coding tasks during the early phases of RLHF training significantly boosts performance, given that these tasks naturally encode fine-grained response distinctions and possess clearly defined ground truths. Through experiments conducted on both small and large models, we demonstrate the effectiveness and scalability of our proposed methods. Our approach exhibits robust generalization capabilities, especially on challenging and out-of-distribution tasks, while yielding significant improvements in mathematics-intensive (STEM) and coding domains. Moreover, the proposed strategies enable the model to effectively capture subtle, task-specific distinctions during the RLHF process, substantially enhancing overall model performance. This work emphasizes the critical role of careful data construction and provides practical methodologies for addressing key performance bottlenecks in RLHF."

Broader Impact

Coverage of Tasks Requiring Long-Form Outputs:

While the current study primarily focuses on tasks involving short-form outputs, we have initiated experiments targeting long-form content generation. In our forthcoming paper, we employ the same methodological framework to address long-text generation tasks. Initial findings are promising, indicating that our approach scales well to long-cot setting. We believe that integrating these additional evaluations will provide a more comprehensive understanding of the applicability and robustness of our method in generating extensive textual content.

Additional Remarks:

Our approach shares similarities with concurrent work, specifically Deepseek-R1, in several aspects. Both methodologies involve initial training on verifiable tasks such as code and mathematics, followed by a preference learning phase. However, our research uniquely identifies and demonstrates the efficacy of this approach in the context of short-text generation. This distinction underscores the versatility and potential of our method across different text lengths and complexities.

Thank you for the detailed response. I appreciate the clarifications.

I will increase my score by one point. However, I still believe that the lack of theoretical support and novelty remains a concern.

Reproducibility and Use of Standard Public Benchmarks

We appreciate the importance of using standard public benchmarks and have indeed conducted evaluations on several open benchmarks. Our results show:

• Significant improvement in coding tasks (e.g., LiveCodeBench: 30.3 -> 36.2).
• Slight improvements or comparable performance in general knowledge tests (e.g., MMLU: 87.3 -> 87.3, MMLU pro: 79.6 -> 80.1).
• Slightly lower but comparable performance on simple math tasks (e.g., AMC23: 68.8 -> 68.7, MATH500: 88.2 -> 87.8).
• Comparable performance on high-difficulty math tasks (e.g., AIME: 18.6 -> 17.9, 26.9 -> 27.2).

However, we've identified limitations in many public benchmarks:

• Potential IID issues, not fully reflecting real-world challenges.
• Risk of data leakage due to widespread use.
• Insufficient difficulty to comprehensively assess advanced models.

For instance, our model can score nearly 90 points on some math benchmarks, but only around 30 points on our more challenging internal test sets.

Consequently, we developed an internal evaluation suite with the following advantages:

• Higher difficulty, better differentiating top-tier model capabilities.
• Continuous iteration, preventing model overfitting.
• Black-box testing, ensuring fair and authentic evaluation.

We believe combining public benchmarks with rigorous internal assessments provides a more comprehensive and reliable evaluation of model performance. We will continue to monitor and use public benchmarks while refining our internal evaluation system.

Regarding the use of open-source models and datasets, we acknowledge their value in enhancing reproducibility. We are considering conducting supplementary experiments with these resources and will include them in future work to provide a more comprehensive comparison.

Experimental Design and Clarity

We appreciate the request for more clarity on our experimental setup and exploration of alternative approaches. While computational constraints limited our ability to exhaustively test all possible configurations, we did conduct several key experiments to explore task ordering and data proportions:

1. On our 25B model, we tested alternative task orderings by: a) Training exclusively on coding data first b) Training exclusively on math data first c) Using our current mixed approach

Results on our internal test set V1 showed:

• Coding first: 38.1
• Math first: 37.8
• Current mixed approach: 38.8

This demonstrates that our current mixed approach outperforms exclusively prioritizing either coding or math tasks.

2. We also experimented with data composition by incorporating all verifiable reasoning data into the math dataset. This yielded nearly identical overall performance (50.6 vs 50.8), suggesting our current data balance is effective.

3. While we didn't test placing math and coding at the very end of training due to computational constraints, we explored a similar concept in subsequent long-chain-of-thought reinforcement learning experiments. These showed comparable performance on open reasoning benchmarks.

These experiments provide evidence supporting our current task ordering and data balance. However, we acknowledge that further systematic exploration could yield additional insights. We're open to conducting more extensive experiments in future work, including generating per-task learning curves for our Early Training Emphasis strategy, as computational resources allow.

In addition, our subsequent paper explains more details on how we implement GenRM and the specifics of PPO training. Due to anonymity policy issues, we cannot share these details at present. The final version will reference this paper.

Q：What if the GenRM is re-trained for several iterations during the RL process? Will that alleviate the reward hacking problem?

A：We have indeed experimented with re-training GenRM for several iterations during the RL process. This approach showed some effectiveness in alleviating the reward hacking problem, and many of our current discoveries still hold true. However, we found that this method requires significant human resources and computational power. Due to these high costs, we ultimately decided against implementing it in large-scale updates and enterprise scenarios. While it offers benefits, the trade-off in terms of resource allocation and efficiency led us to explore other, more scalable solutions for our production environments.

Broader Impact

Coverage of Tasks Requiring Long-Form Outputs:

While the current study primarily focuses on tasks involving short-form outputs, we have initiated experiments targeting long-form content generation. In our forthcoming paper, we employ the same methodological framework to address long-text generation tasks. Initial findings are promising, indicating that our approach scales well to long-cot setting. We believe that integrating these additional evaluations will provide a more comprehensive understanding of the applicability and robustness of our method in generating extensive textual content.

Additional Remarks:

Our approach shares similarities with concurrent work, specifically Deepseek-R1, in several aspects. Both methodologies involve initial training on verifiable tasks such as code and mathematics, followed by a preference learning phase. However, our research uniquely identifies and demonstrates the efficacy of this approach in the context of short-text generation. This distinction underscores the versatility and potential of our method across different text lengths and complexities.

Evaluation and Impact

We sincerely thank the reviewers for their insightful feedback and constructive suggestions. Below, we address the specific questions and concerns raised:

Generalization to Open-Ended Tasks like Dialogue or Creative Writing ： We have conducted manual evaluations on creative writing tasks to assess the generalizability of our approach. The results indicate that our model maintains a consistent performance level comparable to the baselines. Similarly, in dialogue tasks, our model demonstrates performance that is on par with existing methods. These preliminary evaluations suggest that our approach is capable of handling more open-ended tasks effectively. However, we acknowledge that further extensive evaluations are necessary to thoroughly establish the generalization capabilities across diverse open-ended scenarios.

Additional Remarks: Our approach shares similarities with concurrent work, specifically Deepseek-R1, in several aspects. Both methodologies involve initial training on verifiable tasks such as code and mathematics, followed by a preference learning phase. However, our research uniquely identifies and demonstrates the efficacy of this approach in the context of short-text generation. This distinction underscores the versatility and potential of our method across different text lengths and complexities.

We appreciate the reviewers' suggestions to explore the generalization of our approach further. Building on this feedback, we are committed to expanding our evaluations to encompass a wider range of tasks, including more complex and lengthy text generation scenarios. These future endeavors will aim to validate and potentially enhance the applicability of our method in various real-world applications.

Reproducibility and Use of Standard Public Benchmarks

We appreciate the importance of using standard public benchmarks and have indeed conducted evaluations on several open benchmarks. Our results show:

• Significant improvement in coding tasks (e.g., LiveCodeBench: 30.3 -> 36.2).
• Slight improvements or comparable performance in general knowledge tests (e.g., MMLU: 87.3 -> 87.3, MMLU pro: 79.6 -> 80.1).
• Slightly lower but comparable performance on simple math tasks (e.g., AMC23: 68.8 -> 68.7, MATH500: 88.2 -> 87.8).
• Comparable performance on high-difficulty math tasks (e.g., AIME: 18.6 -> 17.9, 26.9 -> 27.2).

However, we've identified limitations in many public benchmarks:

• Potential IID issues, not fully reflecting real-world challenges.
• Risk of data leakage due to widespread use.
• Insufficient difficulty to comprehensively assess advanced models.

For instance, our model can score nearly 90 points on some math benchmarks, but only around 30 points on our more challenging internal test sets.

Consequently, we developed an internal evaluation suite with the following advantages:

• Higher difficulty, better differentiating top-tier model capabilities.
• Continuous iteration, preventing model overfitting.
• Black-box testing, ensuring fair and authentic evaluation.

We believe combining public benchmarks with rigorous internal assessments provides a more comprehensive and reliable evaluation of model performance. We will continue to monitor and use public benchmarks while refining our internal evaluation system.

Regarding the use of open-source models and datasets, we acknowledge their value in enhancing reproducibility. We are considering conducting supplementary experiments with these resources and will include them in future work to provide a more comprehensive comparison.

Theoretical Justification and Novelty

While there isn't much theoretical analysis, in the "Further Experimental Analysis" appendix, we've made some relevant analyses and interpretations.

1. Analyzing prompt filtering by Pre-PPO strategy We collected five responses per prompt, calculated the maximum edit distance among them, and categorized prompts into bins. Then we computed the average normalized reward model score for each bin. Edit distance can reflect the granularity of response differences (larger for coarser-grained, smaller for finer-grained).
2. Observations from the analysis Task-type differences: For tasks supervised by GenRM with ground truth (e.g., math and logic) and without ground truth (e.g., creative writing and cosplay), the normalized reward model scores show different trends as edit distance changes. The model captures coarse-grained differences easily for tasks without ground truth, and is more sensitive to fine - grained ones for tasks with ground truth. Effect of Pre-PPO exclusion: In the Pre-PPO strategy, we excluded certain prompts. An ablation study found that reintroducing previously excluded math and logic prompts improved overall performance marginally. This implies that learning coarse-grained patterns from creative writing and cosplay tasks can harm the scalability of RLHF data. Hypothesis on early task emphasis: Emphasizing math and coding tasks in early training may help the model capture fine-grained distinctions first, reducing the negative impact of learning coarse-grained patterns too early.
3. Reward model analysis Analyzing reward score differences across prompt bins, we found that GenRM with ground truth and RTV assign larger score differences in bins with smaller edit distances. GenRM without ground truth fails to show meaningful score differences in these bins. RTV shows greater ability to capture subtle response distinctions compared to GenRM with ground truth at low edit distances. Overall, reward models with ground truth or verification feedback are more sensitive to fine-grained response variations.